Apparatus and method for 3D surface measurement

ABSTRACT

An apparatus for 3D surface measurement of a target surface, the apparatus comprising: a first projector configured to project a fringe pattern onto the target surface; a second projector configured to project a fringe pattern onto the target surface; a first camera configured to capture the fringe patterns projected by the first projector and the second projector; a second camera configured to capture the fringe patterns projected by the first projector and the second projector; and a computer configured to perform fringe pattern processing of the fringe patterns captured by the first camera and the second camera and to perform data stitching and merging to obtain a 3D surface reconstruction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a filing under 35 U.S.C. 371 as the National Stageof International Application No. PCT/SG2012/000386, filed Oct. 17, 2012,entitled “APPARATUS AND METHOD FOR 3D SURFACE MEASUREMENT”, which claimsthe benefit of and priority to U.S. Provisional Patent Application No.61/548,447, filed Oct. 18, 2011 and entitled “APPARATUS AND METHOD FOR3D SURFACE MEASUREMENT”, both of which are incorporated herein byreference in their entirety for all purposes.

TECHNICAL FIELD

This invention relates generally to an apparatus and method forthree-dimensional (3D) surface measurement, and relates moreparticularly, though not exclusively, to an apparatus and method forthree-dimensional (3D) surface measurement of a non-contactable objecthaving steep profiles.

BACKGROUND

Fringe projection profilometry (FPP) is an optical method for measuringa 3D shape or surface of an object, with the advantages of having a fullfield, high speed and non-contact nature [3-6]. This technique utilizesa projector to project predefined images, which are usually fringepatterns, onto a target surface, and uses a camera to capture thereflected fringe patterns. It can be automatically controlled bycomputer, and recorded images are processed with algorithms to calculatesome intermediate results such as phase values, modulation values, andheight values. Output using FPP may be in the form of a cloud of 3Dpoints, otherwise known as a “3D point cloud.” A triangle mesh orsurface drawings may be applied for better visualization. However,problems of shadow and obstruction issues arise when mechanical movementor rotation, either of the object to be measured or of the projector andcamera, cannot or is preferred not to take place during measurement.

SUMMARY

An apparatus and method for 3D shape measurement is proposed thatenables two projectors and two cameras to form four optical noncontact3D shape sensors for shape measurement and inspection with enhancementof data density and validity. The four optical 3D sensors are located atpositions surrounding a target surface so as to be able to obtain dataof the target surface from four different views without any mechanicalshifting or rotation required of either the target surface or theoptical 3D sensors.

Calibration of all four optical 3D sensors is performed with a sameworld or global coordinates, which makes data combination and mergingeasy without requiring any additional feature points for capturing andidentification.

A software provided for system control and data processing with highautomaticity renders the apparatus and method especially user-friendly.

An algorithm of phase calculation with invalidity identificationframework provides more reliable phase results for dimensional datacalculation.

According to a first aspect, there is provided an apparatus for 3Dsurface measurement of a target surface, the apparatus comprising: afirst projector configured to project a fringe pattern onto the targetsurface; a second projector configured to project a fringe pattern ontothe target surface; a first camera configured to capture the fringepatterns projected by the first projector and the second projector; asecond camera configured to capture the fringe patterns projected by thefirst projector and the second projector; and a computer configured toperform fringe pattern processing of the fringe patterns captured by thefirst camera and the second camera and to perform data stitching andmerging to obtain a 3D surface reconstruction.

The first projector, the second projector, the first camera and thesecond camera may be calibrated with a same global coordinates.

The first projector and the first camera may form a first optical 3Dsensor, the second projector and the first camera may form a secondoptical 3D sensor, the first projector and the second camera may form athird optical 3D sensor, and the second projector and the second cameramay form a fourth optical 3D sensor.

The apparatus may further comprise a frame configured to support andposition the first projector, the second projector, the first camera andthe second camera over and around the target surface.

The first projector and the second projector may be positioneddiametrically opposite each other about the target surface.

The first camera and the second camera may be positioned diametricallyopposite each other about the target surface.

According to a second aspect, there is provided a method for 3D surfacemeasurement of a target surface, the method comprising the steps of: afirst projector projecting a fringe pattern onto the target surface; afirst camera capturing the fringe pattern projected by the firstprojector; a second camera capturing the fringe pattern projected by thefirst projector; a second projector projecting a fringe pattern onto thetarget surface; the first camera capturing the fringe pattern projectedby the second projector;

the second camera capturing the fringe pattern projected by the secondprojector; and a computer processing the captured fringe patterns andperforming data stitching and merging to obtain a 3D surfacereconstruction.

The method may further comprise calibrating the first projector, thesecond projector, the first camera and the second camera with a sameglobal coordinates prior to the first projector projecting the fringepattern onto the target surface.

Processing the captured fringe patterns may comprise performing phaseretrieval and a phase invalidity identification process.

Performing data stitching and merging may comprise calculating aninterval distance between two neighboring points obtained fromprocessing the captured fringe patterns.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the invention may be fully understood and readily put intopractical effect there shall now be described by way of non-limitativeexample only exemplary embodiments, the description being with referenceto the accompanying illustrative drawings.

In the drawings:

FIG. 1 is a schematic illustration of an apparatus for 3D surfacemeasurement;

FIG. 2 is a photograph of the apparatus of FIG. 1;

FIG. 3a is a schematic illustration of projector and camera combinationsprovided to solve shadow and obstruction problems;

FIG. 3b are images obtained by the combinations of FIG. 3 a;

FIG. 4a are phase shifted fringe patterns;

FIG. 4b is a retrieved wrapped phase map;

FIG. 5 is a flowchart of a phase invalidity identification method;

FIG. 6 is an illustration of calibration using a stereovision methodtaking into account camera and projector lens distortion;

FIG. 7 is a collection of different views of a reconstructed 3D surface;

FIG. 8 is an illustration of data registration of a 3D surface;

FIG. 9 is a data visualization of a 3D surface in triangle surfaces withlighting; and

FIG. 10. is a flowchart of an exemplary method for 3D surfacemeasurement.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

An exemplary apparatus 10 and method (100) for 3D surface measurementwill be described with reference to FIGS. 1 to 10 below.

As shown in FIGS. 1 and 2, the apparatus 10 comprises two projectors, afirst projector 22 a and a second projector 22 b connected with agraphic processing unit which is able to expand a virtual computerscreen. The projectors 22 a, 22 b each have a spatial light modulator(SLM) and are configured to project fringe patterns onto a targetsurface 90, serving as light sources for structured light illumination.The SLM is configured to modulate the intensity of the light beamprojected by the projector 22 a, 22 b, and can modulate the phase orboth intensity and phase simultaneously.

The apparatus 10 also comprises two cameras, a first camera 24 a and asecond camera 24 b having appropriately chosen lenses. Connecting cablesused may be either USB or FireWire wires. The cameras 24 a, 24 b eachpreferably have a digital photo sensor that may be a charge coupleddevice (CCD) or a complementary metal-oxide-semiconductor (CMOS) device,with a minimum a resolution of 1,024 by 1,280 pixels.

The projectors 22 a, 22 b and cameras 24 a, 24 b are positioned tosurround the target surface 90, and are connected to a computer 26 witha display serving as a controlling and computing unit. The computer isused to control the projection and capture processes during measurement,and to process the captured images and dimensional data. The performanceof each SLM is controlled by its control unit and the computer 26. Theprojectors 22 a, 22 b can thus be controlled to project any patternsgenerated by the computer 26. In this way, a phase shifting techniquewith temporal phase unwrapping technique can be readily implementedusing these projectors 22 a, 22 b with no mechanical movement requiredof the target surface 90 or the projectors 22 a, 22 b and/or cameras 24a, 24 b.

A frame 26 may be provided to support and position the projectors 22 a,22 b and cameras 24 a, 24 b over and around the target surface 90. Theframe 26 is preferably of a cruciform shape, with the projectors 22 a,22 b and cameras 24 a, 24 b disposed separately at the end of each armof the cruciform shape. In an exemplary embodiment, the two projectors22 a, 22 b are disposed diametrically opposite each other about thetarget surface 90; likewise the two cameras 24 a, 24 b are disposeddiametrically opposite each other about the target surface 90. The frame26 is preferably configured to allow adjustment of the relativepositions of the projectors 22 a, 22 b and cameras 24 a, 24 b in orderto achieve a best configuration for working the apparatus 10.

Using the apparatus 10, a method (100) for true 3D shape measurement ofsurface with fringe projection without mechanical movement is achieved,while solving the shadow and obstruction issues. In the present method(100), the two projectors 22 a, 22 b and two cameras 24 a, 24 b are ableto form four distinct optical 3D sensors using a combination of any oneprojector 22 a or 22 b and one camera 24 a or 24 b as shown in FIG. 3.The four optical 3D sensors thus comprise the following fourcombinations of projectors and cameras:

-   -   Combination 1: projector 22 a with camera 24 a;    -   Combination 2: projector 22 b with camera 24 a;    -   Combination 3: projector 22 a with camera 24 b; and    -   Combination 4: projector 22 b with camera 24 b.

With control by the computer 26, fringe patterns can be projected byeach of the projectors 22 a or 22 b to let each of the cameras capture24 a or 24 b the deformed fringe patterns which contain dimensionalinformation of the tested or target surface 90. Because each of the fourcombinations of optical 3D sensor is located at a different viewingposition, each combination thus gives a 3D point cloud from a differentview. The whole surface shape of the target surface 90 without shadowand obstruction problems can thus be obtained by combining the four 3Dpoint clouds obtained from the four different views by the fourcombinations of optical 3D sensors. All these processes are preferablyconfigured to be done automatically with the computer 26 performing thecontrolling and numerical calculations.

Fringe Pattern Projection and Capture

In the present method (100), each combination 1, 2, 3 and 4 of theoptical 3D sensor performs fringe projection profilometry in which afringe pattern is sequentially projected onto the test object or targetsurface 90 (102) using each of the projectors 22 a or 22 b. Theprojected pattern in this process is preferably predefined. In thepresent method (100), multi-frequency sinusoidal fringe patterns withphase stepping are preferred as they can be easily implemented by thedigital projectors 22 a, 22 b and processed with phase shifting andunwrapping algorithms for phase retrieval subsequently.

In an exemplary embodiment of the method (100), the fringe pattern usedis a sinusoidal fringe, which is usually designed asf=255×(0.5+0.5 cos φ)  (1)

Normally, the phase φ is linearly distributed along one direction.

From a camera view, the projected fringe pattern will seem to bedeformed according to height variation of the tested or target surface90. The fringe pattern projected by each projector 22 a, 22 b isseparately captured by each of the cameras 24 a, 24 b (104), and thecaptured fringe patterns as shown in FIG. 3b are stored in the computer26. Since all four combinations of the optical 3D sensors perform thefringe projection profilometry, four different fringe patterns arecaptured. FIG. 4a is another example of four captured images of a sametarget surface obtained using the apparatus 10.

Generally the captured fringe intensity I can be expressed asI=A+B cos φ,  (2)where A is the background intensity and B is the amplitude of thesinusoidal fringe.Phase Retrieval with Invalidity Identification Framework

Digital image processing then follows, but since the fringe pattern is aspecial type of image, usually the procedure is called fringe patternprocessing or phase retrieval (106). The captured fringe patterns arephase shifted phase patterns as shown in FIG. 4a , and are used as inputdata for a phase shifting algorithm to calculate fringe phase andmodulation.

For phase retrieval, once the phase shifting technique is utilized toobtain a retrieved wrapped phase map as shown in FIG. 4b , the nth phaseshifted fringe patterns I_(n) could be represented by the followingequation (3):I _(n) =A+B cos(φ+δ_(n)),  (3)where the phase shifting amount is calculated by equation (4)

$\begin{matrix}{{\delta_{n} = {2\pi\frac{n}{N}}},{n = 0},1,2,\ldots\mspace{14mu},{N - 1},} & (4)\end{matrix}$and N is the total phase shifting number.

The phase shifting algorithm can be applied for wrapped phase retrieval.The fringe phase result of each fringe frequency is used together tounwrap phase wraps, which limits the phase value to within [−pi, pi] dueto an arctangent operation.

Using a least squares fitting of phase shifting and temporal phaseunwrapping process, not only can the phase be unwrapped, but phasevalidity can be also identified. The wrapped phase φ^(w) and fringemodulation M can be calculated by the following equations (5) and (6):

$\begin{matrix}{{\phi^{w} = {{- \arctan}\frac{\sum\limits_{n = 0}^{N - 1}{I_{n}\sin\;\delta_{n}}}{\sum\limits_{n = 0}^{N - 1}{I_{n}\cos\;\delta_{n}}}}},} & (5) \\{M = {\frac{2}{N}\sqrt{\left( {\sum\limits_{n = 0}^{N - 1}{I_{n}\sin\;\delta_{n}}} \right)^{2} + \left( {\sum\limits_{n = 0}^{N - 1}{I_{n}\cos\;\delta_{n}}} \right)^{2}}}} & (6)\end{matrix}$

Furthermore, the wrapped phase map as shown in FIG. 4b will be unwrappedby using temporal phase unwrapping technique if the multi-frequencyfringe patterns are designed and projected.

The least squares fitting of the unwrapped phase with different fringefrequencies from temporal phase unwrapping will improve the phasemeasuring precision and at the same time it can also provide Root MeanSquare Error (RMSE) as a judging quantity for identification of invalidphase points. A common temporal phase unwrapping procedure is given bythe following equation (7):

$\begin{matrix}{\phi_{k}^{u} = \left\{ {\begin{matrix}{\phi_{k}^{w},} & {k = 1} \\{{\phi_{k}^{w} + {{{{Round}\left( \frac{{\phi_{k - 1}^{u} \cdot {m_{k}/m_{k - 1}}} - \phi_{k}^{w}}{2\pi} \right)} \cdot 2}\pi}},} & {k > 1}\end{matrix},} \right.} & (7)\end{matrix}$and RMSE can be calculated by the following equation (8):

$\begin{matrix}{{RMSE} = {\sqrt{\frac{\sum\limits_{k = 1}^{K}\left( {\phi_{k}^{u} - {m_{k} \cdot X}} \right)^{2}}{K}} = {\sqrt{\frac{\sum\limits_{k = 1}^{K}\left( {\phi_{k}^{u} - {m_{k} \cdot {\sum\limits_{k = 1}^{K}{\phi_{k}^{u}{m_{k}/{\sum\limits_{k = 1}^{K}m_{k}^{2}}}}}}} \right)^{2}}{K}}.}}} & (8)\end{matrix}$

Using fringe modulation M and fitting error RMSE, the invalid phasevalues can be easily identified to improve the reliability of the phasedataset. The whole framework is shown in FIG. 5. Using the phaseinvalidity identification method or framework of FIG. 5 cansignificantly improve the reliability of phase values as well as the 3Dpoint clouds, since the phase invalidity identification process is tomake sure all phase values that will be processed in subsequentdimensional calculations are accurate and reliable.

System Calibration with Global Coordinates

Before actual measurement using the four combinations of optical 3Dsensors is performed, calibration of the system or apparatus 10 shouldbe carried out in advance (101). Calibration (101) of the apparatus 10is different from conventional calibration approaches of fringeprojection profilometry because a same coordinate of calibration for themultiple optical 3D sensors is required in order to make a dimensionaldataset from every combination 1, 2, 3 and 4 of optical 3D sensor havethe same world or global coordinates.

During calibration (101), as shown in FIG. 6, a camera calibrationpattern is presented at different positions and angles within ameasuring range of the apparatus 10. A stereovision calibration methodis then applied to each camera 24 a or 24 b and projector 22 a or 22 bcombination 1, 2, 3 and 4 by treating the projector 22 a or 22 b as acamera according to the fringe phase values. To do so, the calibrationpattern is located at a common position where every projector 22 a, 22 band camera 24 a, 24 b can actually ‘see’ the whole calibration pattern.In this way, common or global coordinates can be determined and thewhole calibration is based on these global coordinates.

Calibration (101) is essential for this apparatus 10 and method (100).The four optical 3D sensors 1, 2, 3, 4 should be calibrated with thesame global coordinates in order to make data stitching in measurementmore convenient. From the phase value, 3D data can be reconstructed oncethe apparatus 10 is calibrated. Ultimately, four sets of dimensionaldata from the four different views obtained by the four combinations 1,2, 3 and 4 of optical 3D sensors are stitched and merged into onedataset (108).

Dimensional Reconstruction with Data Stitching and Merging

Once the system is calibrated, the relationship between phase value anddimensional value is already determined. That is to say, in measurement,if a reliable phase value can be retrieved from the fringe patterns, theout-of-plane and in-plane dimensions can be easily reconstructed withsystem parameters obtained from calibration. For each combination 1, 2,3, or 4 of projector 22 a or 22 b and camera 24 a or 24 b, the resultantdata is one set of dimensional data. In all, the method (100)reconstructs four dimensional datasets as shown in FIG. 7, each datasetcomprising a 3D point cloud. Subsequently, these are combined withmerging of some data at the common regions if the 3D point cloud is toodense, which is not always good for later processing such as datavisualization. As mentioned above, the four datasets are located in thesame global coordinates, as determined during the calibration (101)described above. Therefore, data stitching (108) is relatively easilyperformed, as shown in FIG. 8. The data stitching or merging process(108) is implemented by calculating an interval distance between twoneighboring points in order to remove some too dense points. Theresultant 3D point cloud is friendly for following or subsequentprocessing, if any is desired. For example, as shown in FIG. 9, the 3Ddata can be nicely visualized with triangle surfaces and lighting.

It is envisaged that the above described apparatus 10 and method (100)may be adapted to many production lines for online inspection ofproduced parts. The apparatus 10 may be adjusted to change the measuringvolume for different applications where necessary.

Whilst there has been described in the foregoing description exemplaryembodiments of the present invention, it will be understood by thoseskilled in the technology concerned that many variations in details ofdesign, construction and/or operation may be made without departing fromthe present invention. For example, while two projectors and two camerashave been described above, the apparatus may comprise more than twoprojectors and more than two cameras for more robust 3D surfacemeasurement.

The invention claimed is:
 1. An apparatus for 3D surface measurement ofa target surface, the apparatus comprising: only two projectors, the twoprojectors comprising, a first projector configured to project a fringepattern onto the target surface and; a second projector configured toproject a fringe pattern onto the target surface; only two cameras, thetwo cameras comprising, a first camera configured to capture the fringepatterns projected by the first projector and the second projector and;a second camera configured to capture the fringe patterns projected bythe first projector and the second projector; wherein the firstprojector, the second projector, the first camera and the second cameraare calibrated with a same global coordinates, wherein the firstprojector and the first camera form a first optical 3D sensor configuredto capture a first fringe pattern, wherein the second projector and thefirst camera form a second optical 3D sensor configured to capture asecond fringe pattern, wherein the first projector and the second cameraform a third optical 3D sensor configured to capture a third fringepattern, and wherein the second projector and the second camera form afourth optical 3D sensor configured to capture a fourth fringe pattern;and a computer configured to: perform fringe pattern processing of thefirst fringe pattern to generate a first 3D point cloud, perform fringepattern processing of the second fringe pattern to generate a second 3Dpoint cloud, perform fringe pattern processing of the third fringepattern to generate a third 3D point cloud, perform fringe patternprocessing of the fourth fringe pattern to generate a fourth 3D pointcloud, and perform data stitching and merging of the first 3D pointcloud, the second 3D point cloud, the third 3D point cloud, and thefourth 3D point cloud to obtain a 3D surface reconstruction using thecalibration with the same global coordinates.
 2. The apparatus of claim1, further comprising a frame configured to support and position thefirst projector, the second projector, the first camera and the secondcamera over and around the target surface.
 3. The apparatus of claim 1,wherein the first projector and the second projector are positioneddiametrically opposite each other about the target surface.
 4. Theapparatus of claim 1, wherein the first camera and the second camera arepositioned diametrically opposite each other about the target surface.5. A method for 3D surface measurement of a target surface, the methodcomprising the steps of: a first projector, of only two projectors,projecting a fringe pattern onto the target surface; a first camera, ofonly two cameras, capturing a first fringe pattern projected by thefirst projector; a second camera, of the only two cameras, capturing asecond fringe pattern projected by the first projector; a secondprojector projecting a fringe pattern onto the target surface; the firstcamera capturing a third fringe pattern projected by the secondprojector; the second camera capturing a fourth fringe pattern projectedby the second projector; wherein the first projector and the firstcamera form a first optical 3D sensor, wherein the second projector andthe first camera form a second optical 3D sensor, wherein the firstprojector and the second camera form a third optical 3D sensor, andwherein the second projector and the second camera form a fourth optical3D sensor; a computer processing the captured fringe patterns andperforming data stitching and merging to obtain a 3D surfacereconstruction, wherein processing the captured fringe patternscomprises processing the first fringe pattern to generate a first 3Dpoint cloud, processing the second fringe pattern to generate a second3D point cloud, processing the third fringe pattern to generate a third3D point cloud, and processing the fourth fringe pattern to generate afourth 3D point cloud; and calibrating the first projector, the secondprojector, the first camera and the second camera with a same globalcoordinates prior to the first projector projecting the fringe patternonto the target surface.
 6. The method of claim 5, wherein processingthe captured fringe patterns comprises performing phase retrieval and aphase invalidity identification process.
 7. The method of claim 5,wherein performing data stitching and merging comprises calculating aninterval distance between two neighboring points obtained fromprocessing the captured fringe patterns.